<html>
<head><meta charset="utf-8"><title>coercions-debug noise · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html">coercions-debug noise</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="239275491"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275491" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275491">(May 18 2021 at 15:40)</a>:</h4>
<p>I ask because this week, several PR’s had a (small, ~1.4%) performance impact when fully compiling coercions-debug</p>



<a name="239275528"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275528" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275528">(May 18 2021 at 15:40)</a>:</h4>
<p>and I’m just musing about how much of that could be measurement noise</p>



<a name="239275622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275622">(May 18 2021 at 15:41)</a>:</h4>
<p>but if it were truly noise, then i’d expect to see numbers going <em>up</em> on occasion as well</p>



<a name="239275687"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275687" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275687">(May 18 2021 at 15:41)</a>:</h4>
<p>coercions-debug is pretty typically noise</p>



<a name="239275699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275699">(May 18 2021 at 15:41)</a>:</h4>
<p>you can look at graph view</p>



<a name="239275796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275796">(May 18 2021 at 15:42)</a>:</h4>
<p><a href="/user_uploads/4715/g8mXa87iaQwV0sSemtHf-Ioy/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/g8mXa87iaQwV0sSemtHf-Ioy/image.png" title="image.png"><img src="/user_uploads/4715/g8mXa87iaQwV0sSemtHf-Ioy/image.png"></a></div>



<a name="239275821"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275821" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275821">(May 18 2021 at 15:42)</a>:</h4>
<p>oh right, I guess I wouldn’t see the “going up” cases while I’m reviewing the regressions</p>



<a name="239275864"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275864" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275864">(May 18 2021 at 15:42)</a>:</h4>
<p>I guess I should read the whole report before making my notes</p>



<a name="239275916"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275916" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275916">(May 18 2021 at 15:42)</a>:</h4>
<p>in general coercions-debug is almost always ignorable</p>



<a name="239275926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275926">(May 18 2021 at 15:43)</a>:</h4>
<p>or maybe the script could detect when a single benchmark is bouncing around during the week?</p>



<a name="239275976"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275976" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275976">(May 18 2021 at 15:43)</a>:</h4>
<p>well, that's likely hard, or at least, 'bouncing around' is generally an unknown to me</p>



<a name="239275990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239275990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239275990">(May 18 2021 at 15:43)</a>:</h4>
<p>and you may want to know if it is doing that, anyway</p>



<a name="239276030"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276030" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276030">(May 18 2021 at 15:43)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/triage.202021.2005.2018/near/239275916">said</a>:</p>
<blockquote>
<p>in general coercions-debug is almost always ignorable</p>
</blockquote>
<p>I don’t doubt this is true, but my gut reaction to it is: “Then we should either drop it from the benchmarks, or change what we are measuring for it."</p>



<a name="239276033"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276033" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276033">(May 18 2021 at 15:43)</a>:</h4>
<p>in the case of coercions-debug we can add another special case for it to the script; it's already special cased on the incr-patched side</p>



<a name="239276183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276183">(May 18 2021 at 15:44)</a>:</h4>
<p>probably someone should figure out why only coercions-<strong>debug</strong> is noisy</p>



<a name="239276209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276209">(May 18 2021 at 15:44)</a>:</h4>
<p>check and opt builds are much less so, at roughly 0.5% over past week, for example</p>



<a name="239276238"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276238" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276238">(May 18 2021 at 15:45)</a>:</h4>
<p><a href="/user_uploads/4715/Yl-ZtNpPSsM_5Jd6NvcRpdkn/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/Yl-ZtNpPSsM_5Jd6NvcRpdkn/image.png" title="image.png"><img src="/user_uploads/4715/Yl-ZtNpPSsM_5Jd6NvcRpdkn/image.png"></a></div>



<a name="239276285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276285">(May 18 2021 at 15:45)</a>:</h4>
<p><a href="/user_uploads/4715/hoiWxGE2yXL2vrwZd8RxTMEY/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/hoiWxGE2yXL2vrwZd8RxTMEY/image.png" title="image.png"><img src="/user_uploads/4715/hoiWxGE2yXL2vrwZd8RxTMEY/image.png"></a></div>



<a name="239276435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276435">(May 18 2021 at 15:46)</a>:</h4>
<p>I'll actually kick some off now</p>



<a name="239276464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276464">(May 18 2021 at 15:46)</a>:</h4>
<p>wait what are you kicking off?</p>



<a name="239276581"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239276581" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239276581">(May 18 2021 at 15:47)</a>:</h4>
<p>coercions-debug under cachegrind, I'll run it twice with the same compiler and diff the two; my expectation is I'll see some noise there. If not, a compiler rebuild with slight adjustments and the same thing.</p>



<a name="239277471"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239277471" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239277471">(May 18 2021 at 15:52)</a>:</h4>
<blockquote>
<p>well, that's likely hard, or at least, 'bouncing around' is generally an unknown to me</p>
</blockquote>
<p>Maybe just measuring the recent variance or some similar statistical measure (nth percentile of step sizes, since it's a discrete series) and showing that in comparisons would help?</p>



<a name="239277597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239277597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239277597">(May 18 2021 at 15:53)</a>:</h4>
<p>Maybe. I don't quite know how easy those are to implement pull off, or whether they lie on firm grounding.</p>



<a name="239277992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239277992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239277992">(May 18 2021 at 15:55)</a>:</h4>
<p>Definitely not opposed to discussing on an issue or seeing an implementation; I'd love to have some more principled change detection. I've been looking into it in the little free time I have over the past year and hadn't come up with anything that seemed very reliable.</p>



<a name="239278933"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239278933" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239278933">(May 18 2021 at 15:59)</a>:</h4>
<p>Hm, looks like most of it is due to LLVM, but I'll need to dig in more to gain a better understanding of what's going on that causes such large disparities.</p>



<a name="239281815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239281815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239281815">(May 18 2021 at 16:15)</a>:</h4>
<p>I remember some benchmarking work from <span class="user-mention silent" data-user-id="252006">Laurence Tratt</span>  on virtual machines (pypy, hotspot, etc) a few years back where they used change-point analysis with great success for these kinds of issues. Since the softdev team now works on the yorick meta-tracer in rust, we could probably ask them, e.g. <span class="user-mention silent" data-user-id="223288">Edd Barrett</span>  is a regular here :)</p>
<p>(the 2 articles/papers I remember were: <a href="https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_1.html">https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_1.html</a> and <a href="https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_2.html">https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_2.html</a>)</p>



<a name="239281832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/coercions-debug%20noise/near/239281832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/coercions-debug.20noise.html#239281832">(May 18 2021 at 16:15)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/coercions-debug.20noise/near/239277992">said</a>:</p>
<blockquote>
<p>Definitely not opposed to discussing on an issue or seeing an implementation; I'd love to have some more principled change detection. I've been looking into it in the little free time I have over the past year and hadn't come up with anything that seemed very reliable.</p>
</blockquote>
<p>The last paper I remember reading on this subject was <a href="https://kar.kent.ac.uk/33611/">Rigorous Benchmarking in Reasonable Time</a> by Kalibera and Jones.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>